Two-tier Bloom filter to achieve faster membership testing

Introduction: Bloom filters [1] are a space-efficient, probabilistic data structure for representing a list of elements (for example, a list of strings). A Bloom filter is an array of m bits. A string is mapped into a Bloom filter by inputting it to a group of k hash functions resulting in k array positions. Each indexed array position is set to 1. A string is tested for membership by inputting it to the same group of k hash functions. If all k generated array positions are determined to be set to 1, then the string is probably a member. False positives can occur with Pr [false positive] 1⁄4 (12 1/m) for n elements mapped into a Bloom filter. Bloom filters are widely used with many applications in the domain of networks [2]. One application of interest is representing large file lists; for example lists of shared files in servers or caches to enable determination if a given file name is in a list of shared files. The key performance measures for a Bloom filter are membership testing time (also called computation time in [3]), memory requirements, and probability of false positive. Membership testing time is the time to determine if an element belongs to the set represented by the Bloom filter. The key motivation for using Bloom filters (over more conventional data structures) is reduced memory requirement and faster membership testing [3]. Membership testing time is a function of the time to (a) compute up to k hashes and (b) perform up to k lookups in memory where the Bloom filter array is stored. Memory lookup times depend on the type of memory in which the Bloom filter is stored (e.g. high-speed localised static random access memory (SRAM) or slower main memory dynamic random access memory (DRAM). In this Letter, the membership testing time for a Bloom filter is reduced by implementing hashing directly in specialised hardware and by introducing a second tier cache Bloom filter to reduce the number of accesses required into slower main memory.